Efficient multivariate entropy estimation via k-nearest neighbour distances

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient in the sense of achieving the local asymptotic minimax lower bound. To this end, we initially study a generalisation of the estimator originally proposed by Kozachenko and Leonenko (1987), based on the k-nearest neighbour distances of a sample of n independent and identically distributed random vectors in Rd. When d ≤ 3 and provided k/ log n→∞ (as well as other regularity conditions), we show that the estimator is efficient; on the other hand, when d ≥ 4, a non-trivial bias precludes its efficiency regardless of the choice of k. This motivates us to consider a new entropy estimator, formed as a weighted average of Kozachenko–Leonenko estimators for different values of k. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness. In addition to the new estimator proposed and theoretical understanding provided, our results also have other methodological implications; in particular, they motivate the prewhitening of the data before applying the ∗Research supported by a Ph.D. scholarship from the SIMS fund. †Research supported by an EPSRC Early Career Fellowship and a Philip Leverhulme prize. ‡Research supported by NSF FRG Grant DMS-1265202 and NIH Grant 1-U54AI117924-01. 1 ar X iv :1 60 6. 00 30 4v 2 [ m at h. ST ] 1 4 Ju l 2 01 6 estimator and facilitate the construction of asymptotically valid confidence intervals of asymptotically minimal width.